arXiv: 1502.05142vl [cs.IT] 18 Feb 2015 


1 


Models, Statistics, and Rates of 
Binary Correlated Sources 

Marco Martalo and Riccardo Raheli 


Abstract — This paper discusses and analyzes various models of 
binary correlated sources, which may be relevant in several dis¬ 
tributed communication scenarios. These models are statistically 
characterized in terms of joint Probability Mass Function (PMF) 
and covariance. Closed-form expressions for the joint entropy 
of the sonrces are also presented. The asymptotic entropy rate 
for very large number of sources is shown to converge to a 
common limit for all the considered models. This fact generalizes 
recent results on the information-theoretic performance limit 
of communication schemes which exploit the correlation among 
sources at the receiver. 

Index Terms —Correlated sources, binary sonrces, statistical 
characterization, entropy rate, achievable region. 


I. Introduction 

The efficient transmission of correlated signals, observed at 
various nodes, to one or more collectors is of wide interest 
in various scenarios, such as sensor networks [1], and has 
been the subject of recent research attention. For instance, [2] 
discusses the spatial dependence between data according to 
the distribution of the nodes in the monitored area through 
empirical measurements. The design of efficient transmission 
schemes for correlated sources through orthogonal additive 
white Gaussian noise (AWGN) channels is discussed in [3]. In 
this case, the separation between source and channel coding is 
optimal and ultimate performance can be achieved by means 
of distributed source coding (DSC) followed by independent 
capacity-achieving channel coding [3], [4]. An alternative 
solution is based on the use of joint source channel coding 
(JSCC) schemes, where proper codes are used to encode the 
correlated sources. In both cases, knowledge of the statistical 
source correlation is exploited at the joint decoder, whereas 
source encoding is performed separately [5]. 

The design of universal codes for transmission of correlated 
sources, i.e., capacity-achieving codes for all possible channel 
parameters, is a current topic. Universal codes based on spatial 
coupling have been recently proposed [6]. Orthogonal multiple 
access schemes with an arbitrary number of correlated sources 
have been recently addressed in [7], where the asymptotic 
achievable region for increasing number of sources has been 
characterized in terms of individual channel capacities, for a 
specific correlation model, and pragmatic joint source-channel 
coded schemes have been proposed. 

In this paper, we discuss various correlation models for an 
arbitrary number of binary sources, which may be of interest in 
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several realistic communication scenarios. With the exception 
of [7], there are not many papers in the literature discussing 
correlation models for a possibly large numbers of sources. 
In [8], the authors proposed a correlation model based on 
a set of linear equations in the binary field. This model is 
shown to be related, under special conditions, to one of the 
binary symmetric channels (BSC)-based models discussed in 
this paper. Main contributions of this paper are the statistical 
characterization of such models in terms of joint probability 
mass function (PMF), covariance, and joint entropy of the 
sources. Moreover, considering the information sequence as 
a stochastic process in the spatial domain, we derive the 
asymptotic entropy rate for large number of sources and show 
it coincides for all the considered models. Therefore, the 
asymptotic achievable region discussed in [7], characterized 
in terms of the source entropy rate, can be inferred to be 
independent of the specific correlation model and pragmatic 
joint source-channel coded schemes may be expected to have 
similar asymptotic behavior regardless of the model. 

This paper is structured as follows. In Section|IIj we present 
various binary source correlation models. In Section III we 


statistically characterize these models, by deriving the joint 
PMF of their output sequences and the corresponding co- 
variance matrices. In Section]^ we statistically characterize 
these schemes in terms of their joint entropy. In Section [V] 
we use the joint entropy rate of these models to characterize 
the performance limit of orthogonal multiple access schemes 
transmitting correlated symbols. Finally, concluding remarks 


are given in Section VI 


II. Source Correlation Models 

Consider N source nodes, possibly spatially distributed, 
which output (emit) binary information sequences X — 
{Xi, X 2 , ■ ■ ■, Xn)'^, where is the transpose operation. 
The binary information symbols are assumed to be marginally 
equiprobable, but correlated with each other according to a 
given PMF Px{x), in which the V-element vector x describes 
a possible realization of X. This scenario may be represen¬ 
tative of a sensor network in which the sensors observe N 
correlated physical quantities of interest. In Fig. [1] possible 
correlation models are shown: (a) parallel, (b) serial, and (c) 
mixed. In the parallel model (a), the source symbols are the 
output of a set of parallel BSCs, with cross-over probability 
1 — pi, for I = 1, 2,..., V, denoted as BSC(pf), whose input 
is a hidden common information bit B. The f-th source symbol 
is given by 


Xi = B® Zi 


( 1 ) 
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Fig. 1. Considered correlation models: (a) parallel, (b) serial, and (c) mixed. 


where B is an independent equiprobable binary random vari¬ 
able, {Zi} are independent binary random variables with 
P{Zi^ = 0) = p^, 1/2 < < 1 for i = 1,2, and 

0 denotes a modulo-2 sum. Obviously, if p^ = 0.5 there is no 
correlation among the binary information symbols 
whereas if = 1 they are identical with probability 1. 

In Fig. [T] (b), a possible serial correlation model is shown, 
in which the source symbols are correlated by a cascade of 
BSCH with cross-over probability 1 — p^. Again the output 
symbols are uncorrelated for pg = 0.5, whereas they are equal 
with probability 1 for pg = 1. 

A more general “mixed” case with a number m of serial 
chains is shown in Fig. [T] (c), in which 1 — p^ denotes the 
cross-over probability of the Ath BSC on the j-th chain. In this 
case, the correlated data at the £-th, i = 1,2,..., N, source 
on the j-th branch, j = 1,2,..., M, can be expressed as 

e 

Xgj = 50^ Zij = B® Zgj 
i=l 


where the symbol ^0 denotes modulo-2 sums. The random 

*The first BSC does not play any role in the serial model and could be 
omitted—it is kept for notational consistency with the other models. 


Note that for M = 1, this mixed model reduces to the serial 
one of Fig. (b) and the index j in Q can be dropped. 

A model based on a set of binary linear equations was 
considered in [8]: 

AX (3) 


where A is a binary matrix (whose entries are equal to either 0 
or 1) defining the set of equations and Z = {Zi, Z 2 ,..., ZmY" 
is a vector of independent Bernoulli binary random variables 
with parameter pg = P{Zg = 0). Note that matrix operations 
are performed in the binary field. As a special case, 0 may 
encompass a set of recursive equations of the form 


min{£),r-l} 

2® Ag^g-iXg-i = Zg 


i=0 


(4) 


in which D is the recursion depth, Agg = 1, Ag^ = 0 or 1 for 
k = £ — min{i9, £ — !},...,£— 1, and Agk — 0 for k > £. 
Since we are considering binary random variables, 0 can be 
also rewritten as 


X, = 


Zg 


Zg 


min{£l,r-l} 

0 ^ Agj-iXg-i 

i=l 


£=1 

£ = 2,3,...,N. 


The model in Q describes a recursive binary filter with input 
Zg and output Xg. In the special case of i9 = 1, Ag^g-i = 1, 
and Pi = 1/2, the model in 0 is equivalent to the serial one 
in Fig. [2 (b). 

If the matrix A is invertible in the binary field, then 


X = A~^Z 


( 6 ) 


and, therefore, there exists a one-to-one correspondence be¬ 
tween X and Z. A case where the inverse exists is the 
recursive model in 0, as it is shown in Appendix where a 
few cases of interest are also analyzed. For simplicity, in the 
rest of the paper we will assume that the matrix A is invertible. 
Note that, invertibility may lead to uncorrelated binary sources 
in the special case of matrices with constant row weight equal 
to dc, as shown in [10]. 


III. Statistical Characterization 

According to the parallel correlation model 0 in Fig.[T](a), 
the joint PMF of the information symbols at the output of the 
N nodes can be computed. By straightforward manipulations, 
one can show that 


Px{x) = Y. Px{x\B = b)PB{h) 

b=0,l 
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where 5o and 5i is a partition of the set { 1 , 2 , 
specifying the positions of zeros and ones in x, respectively. 
In the special case of = p, £ = 1,2,N, one obtains 

Pxix) = ^ + (1 - (8) 

where riz is the number of zeros in x. 

In the serial case of Fig. [T](b), using the chain rule, one has 

Px(x) = y] PxjxjB = b)PB{h) 

N 

= Y, PB{h)PxAxi\B^h)\{PxAxAXi-i=xi.i){9) 
6 = 0,1 i =2 

where we have used the fact that, given X^_i, is con¬ 
ditionally independent from the previous source symbols 
Xi- 2 , ■ ■ ■ ,Xi and B. After further simple manipulations, one 
can write 

Pxix) = Pi Wil- Pk) 

teCo keCi 

where Cq and Ci is a partition of the set (2,3, ...,A^} 
specifying the positions where xi = X£-i and xi A X£-\, 
respectively. In the special case of pi = p, £ = 2,,N, 
denoting as the cardinality of the set Cq, one obtains 

Pxix) = \p<{l-p)^-<-\ 

In the general mixed case of Fig. (c), let us denote as 

XA,^{Xi„...,Xmj) 3 = 1,2,...,M 

the set of the N source symbols on the j-th branch. One can 
generalize the result of the serial case by writing: 

Pxix) = Y. Pxix\B = b)PBih) 

6 ^ 0,1 

M 

= YI[PxyxAAB = b)PBib) 

M 

= YI[P^(’^)PxyxiAB = b) 

N 

1=2 


The covariance matrix for the considered correlation models 
is defined as 


Cx — Rx — IJ-xlY- 


For the parallel and serial models, the elements of the corre¬ 
lation matrix Rx and the mean vector Hx are, respectively. 


Rik = V.[XiXk] 

Pi = E[Xi] 


i,k = l,2,...,N. 


Note that Rik = ^[XiXk] = PiXi = Xk = 1), since the data 
are binary. The elements of the vector px can be obtained as 

1 

^,= 2 - 

Rx depends on the considered correlation model, but in all 
cases 

i?,, = E [X^] = 1 • PX = 1) = ^ 

since, for all considered models, PiXi = 1) = 0.5. 

In the parallel case of Fig. [T] (a), for i,k = 1,2,... ,N 
(i A B) one obtains 

E[X,Xfc] = 1 ^ =Xfe = l|B = 6 ) 

^ 6 ^ 0,1 

= ypiPk + A - p^)il - pk)] 

= 2A ~ iPi Pk) +‘^PiPk] (11) 

which reduces, for pi = p, £= 1,2,... ,N, to 
E[X,Xfe] = i [1 - 2 p + 2 p 2 ] . 

As expected, the correlation is independent of the indices of 
the considered sources. Note that if p = 1/2, Cik = 1/4 for 
i = k and zero otherwise, i.e., data are uncorrelated. 

For the serial correlation model in Fig.[T](b) and i 7 ^ fc, one 
can write 


p{x, = Xk = i) = PiXk = iX = i)PX = i) 

= ^PiXk = iX = i) 

where the conditional probability PiXk = l\Xi = 1 ) can be 
computed noting that Xi = Xk = 1 if the BSCs of indices 
i + l,i + 2,... ,k flip an even number of times. Therefore, by 
arguments similar to those in [9, Lemma 4.1] we obtain 


where, in the first line, we have used the fact that, conditionally 
on B, the branches are independent, and the second line is 
equivalent to (|^ applied to each branch. 

Finally, the PMF for correlated sources following the model 
in [ 8 ] can be characterized, by resorting to ([^, as 

Pxix) = PziAx). (10) 

Using simple manipulations, we have 

Pxix) = n n p^) 

l^S'g k^S[ 

where S'q and is a partition of the set (1,2, ...,A^} 
specifying the positions of zeros and ones in Ax, respectively. 


PiXk = iX = 1 ) 


k 

1 + n (2-°^ ■ 

l=i+l 


and, therefore. 


EXXk] 


1 

4 


fc 

1 + n (2-°^ ■ 

l=i+l 


( 12 ) 


For the special case of pi = p, £ = 2,2>,... ,N, denoting the 
number of hops in the chain of BSCs as L = \i — k\, 
reduces to 

E[X,Xfc] = i[l + ( 2 p-l)^]. 






4 



(a) 



Fig. 2. Coefficients of the first row of the covariance matrix Cx for N = 6 sources and three models: (a) parallel, (b) serial, and (c) mixed with M = 2. 
Two values of p are considered: 0.7 (left bars in each figure) and 0.95 (right bars in each figure). 


Note that if p = 1/2, Cik = 1/4 for i = A: and zero otherwise, 
i.e., data are uncorrelated. 

In the mixed scenario of Fig.[T](c), the covariance matrix has 
size NM X NM. The element Cik, for A, fc = 1, 2,..., NM, 
can be defined as the covariance between the source symbols 
and Xi^rn^, where 

i = {mi — l)N + ii k = {m2 — f)N + £ 2 - 


In particular, the following two cases may occur; 


if mi = m 2 , i.e., Xg^mi and Xi^mi belong to the same 
branch, the result in (12 1 can be applied; 
if mi ^ m 2 , i.e., Xe^^i and Xi^ran belong to different 
branches, the results for two sources in a parallel scheme 


can be applied replacing pi and pk 
and pi.^rn- 2 , respectively, as defined 


k in ( [TT] ) wi 
in (|2|^ 


with 


Obviously, it is still verified that, if pik = p = 1/2, Cik = 1/4 
for i = k and zero otherwise, i.e., data are uncorrelated. 

Finally, the covariance matrix can be also considered for the 
correlation model ([^ based on a set of binary linear equations. 
Using (|^, one can write 


Cx = Rx — MxMx 

= E [A~^ZZ'^{A-~'^Y] - E [A-^Z] E [Z^{A--^Y'] 

= P{A-^ZZ^{A-Y = Jn) 

-P {A-^Z = 1^) P {Z^{A~Y = lx) 


where we have used the fact that the mean value of a binary 
random variable is equal to the probability that the variable is 
equal to 1, Jx is the all-one matrix of size N x N, and l^v 
is the all-one column vector of length N. Note that a closed- 
form solution for Cx is not readily available as it depends on 
the particular structure of A. 

Fig. shows the coefficients of the first row of the co- 
variance matrix Cx for N = 5 sources, pi = pik = p, and 
the three models: (a) parallel, (b) serial, and (c) mixed with 
M = 2. Two values of p are considered; 0.7 (left bars in 
each figure) and 0.95 (right bars in each figure). Only the first 
row is considered, since for parallel and serial models with 
constant p the covariance matrix is symmetric and Toeplitz. In 
the mixed case, instead, for constant p, Cx has the following 


block structure 


Cx = 


Cl 

C 2 

C^2 

Cl 

/^T 
^ M 

/^T 

^M-1 


Cm 

Cm-1 

Cl 


where Ci is the covariance matrix, of size N x N, between 
sources on chains with separation i — 1. In other words, 
zero separation means that the sources are in the same chain, 
separation 1 means that sources are on adjacent chains, and so 
on. In this case as well, the first row is sufficient to characterize 
the entire matrix. In the figure, one can observe that the first 
coefficient is equal to the symbol variance (1/4) in all cases. 
Moreover, in the parallel model (a), all coefficients Cik for 
k from 2 to 5 are equal due to the fact that the pairwise 
probabilities are identical, regardless of the source index. Note 
also that the higher the value of p, the higher the covariance 
elements, since data are more and more correlated. In the serial 
model (b), the covariance decreases with k, since a larger 
number of BSCs is present between the sources, which become 
more and more uncorrelated. In the mixed model (c), recall 
that indices from A; = 1 to A; = 5 correspond to the first chain 
of BSCs, whereas indices from A: = 6 to A: = 10 correspond 
to the second chain. As expected, symbols in the second chain 
are less correlated with the first symbol Xu, than those in the 
first one. 

An alternative way to view the covariance matrix is by the 
histogram of its values shown in Fig. for = 5 sources 
and two models: (a) parallel and (b) serial. Two values of 
pi = p are considered; 0.7 and 0.95. One can observe that in 
the parallel model, only two values of Cik are allowed, since 
the pairwise probabilities are the same for any pair of sources. 
Moreover, for high correlation (e.g., p = 0.95) larger values 
than those for small correlation (e.g., p = 0.7) are obtained, 
which is in agreement with the fact that Cik 0 for i k if 

p —>■ 1/2. 


IV. Source Entropy Rate 
To compute the joint entropy of the N sources for the 
considered correlation models, we denote it as H{Xi), where 
the notation X\ indicates the sequence {Xa, Xa+i, ■ ■ ■, Xb). 
Let us consider the joint entropy of X^ and B: 

H{B,X^) = H{B) + H{X^\B). 


(13) 
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Fig. 3. Histogram of the values of the matrix Cx for = 5 sources and 
two models: (a) parallel and (b) serial. Two values of p are considered: 0.7 
and 0.95. 


For the parallel model Q of Fig. (a), one has 

N 

h{x^\b) = J2hm 

e=i 


where is the entropy of a binary random variable 

with parameter Since B is a uniformly distributed binary 
random variable, H{B) = iJb(0.5) = 1. Therefore, one 
obtains: 

N 

H{B,X^) = 1 + Y,HM- 


As in (13 1 , it is also possible to write 

HiB, X^) = HiXf) + HiB\X^) = 1 + ^ HM 


N 


^=1 


and, therefore, 


N 

HiX^) = l + J2HM-H{B\X^). (14) 

e=i 


Combining ( [T5| ) and ( [T6| ) and analyzing the limit for large 
number of sources, one obtains 


lim 

AT—>-+oo 


HjXn 

N 


= H^ 


(17) 


where 


i?b = 


lim 

N—^-\-oq 


1 

N 


N 


e=i 


in which the limit exists since 0 < H\^{pi) < 1. This quantity 
can be interpreted as the limit average entropy of the cascade 
of N BSCs and ( [T7] i can be interpreted as the asymptotic 
entropy rate of the correlated sources, namely the limit average 
source entropy. Equation ( [T7| ) reduces, for the special case 
Pf = p, £= l,2,...,iV, to 

- 1 ^ 

= Hh{p)- 

JV-^+oo JV ' 

1=1 


This result can be also obtained by observing that X^ is a 
stationary binary random process, whose entropy rate is well- 
known [11, Ch. 4]. This limit for the special case of constant 
p has been also derived in [7]. 

Consider now the joint entropy of the serial correlated 
source model in Fig. (b). Using the chain rule for entropy, 
one can easily compute the joint entropy as 


N 

H{X^) = + 

i^2 

N 

= H{X,) + Y,H{Xe\Xe.i) 

i^2 

N 

= l + ^i7b(pP. (18) 

t=2 


Analyzing the limit for large number of sources, one obtains 


lim 

AT—>-+oo 


H{X^) 

N 


= i?b 


(19) 


and in the special case of p^ = p, £ = 2, 3,..., V: 


ffb = 


lim 

N—¥-\-00 


1 

N 


N 

1 + 

1=1 


= i?b(p). 


By definition of entropy, the last term is non negative and the 
following upper bound (UB) is obtained 

N 

H{X^)<1 + Y,HM- ( 15 ) 

e=i 

Moreover, since conditioning reduces entropy [11], it also 
follows that 

H{B\X^) < H{B\Xi) = i7b(pi). 

Using this in ( [T4| >, one obtains the lower bound (LB): 

N N 

H{X^) > 1 + l + (16) 

e=i i=2 


Note that the parallel and serial models have equal asymptotic 
source entropy rate. 

In the general mixed case of Fig. (c), it can be shown 
(see Appendix |II] for the proof) that 


lim 


H{X(i,...,X. 


N \ 

im) 


MN 


= Hy. 


( 20 ) 


where 


i?b = 


lim tttt 
N^+oo MN 


M N 


Hh{ptj)- 


j=i r=i 


We remark that the limit exists since each term in the 
summation is limited to the interval [0,1]. Equation (201 
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reduces, for the special case of pij = p, i = 1,2, ...,N 
and j = 1,2,..., M, to 

_ M N 

Miv S E "M = "M- 

j = l i=l 

Note that similar considerations can be also carried out if N 
is kept fixed and M grows to infinity or if both N and M 
become arbitrarily large. 

We finally analyze the entropy rate of the source correlation 
model given by a set of linear equations in ([^. Using ( [T0| and 
the assumption of invertibility of A, it can be easily shown 
that [11] 

N 

h{x^^) = h{z^) = J2hM- 

e=i 


Therefore: 


lim 

N^-\-oo 


Hjxn 

N 


Hb 


which reduces, for the special case of pi = p, £ = 1,2,..., N, 
to 

_ 1 ^ 

Af —>•+00 ^' 

t=l 


This shows that the source entropy rate is asymptotically the 
same for the linear equation-based correlation model as well, 
hence for all the considered correlation models. 

The convergence of the entropy rates of the considered 
models can be analyzed in terms of the difference between the 
average source entropy for finite N and the asymptotic value. 
In particular, in the parallel and serial models we define 


A Hjxn 

N 


Hb. 


In the mixed model, instead, this difference is also function 
of M and can be defined as 

^ HiX^{,...,X^^) = 

- MN - 


In Fig. g e is shown, as a function of N, for parallel or serial 
models and two values of p (assumed equal for all BSCs): 
0.7 and 0.95. Note that lower and upper bounds on e can 
be obtained using the LB and UB on the joint entropy for 
the parallel model, respectively. In particular the LB for the 
parallel case and the exact value of e for the serial model are 
identical, due to the fact that the right-hand side of and 
coincide for any N in the special case of constant p. 
Moreover, the UB on e for the parallel model is the same, 
regardless of the value of p, and the curves overlap for both 
values of p. In both cases, e converges as 1/iV and the same 
convergence can be obtained for the model given by a set of 
linear equations when A is such that this model is equivalent 
to the serial one. Furthermore, the tightness of the bounds 
increases with the number of correlated sources, since UB and 
LB become closer to each other. However, the convergence of 
the LB degrades with increasing values of p. Moreover, the 
mixed model has the same trend for e, since it decreases with 
1/[MN) for any fixed value of M. 

In Fig. g e is shown, as a function of M, for the mixed 
model, two values of p (namely, 0.7 and 0.95), and two values 



Fig. 4. e, as a function of N, for parallel (par), serial (ser), and mixed (mix) 
(M = 2) models and two values of p: 0.7 and 0.95. 


of N (namely, 10 and 50). Note that similar considerations as 
those relative to Fig. |^hold in this case for fixed values of N 
and letting M go to infinity. 


V. Transmission of Correlated Symbols in 
Orthogonal Multiple Access Schemes 


The computation of the source entropy rate discussed 
in Section plays an important role in determining the 
asymptotic achievable region of orthogonal multiple access 
schemes with correlated sources. In such schemes, each node 
independently encodes, by a joint source-channel code with 
rate r, the source symbols and transmits them through an 
orthogonal multiple access channel. At the receiver side, 
data are decoded by properly taking into account the source 
correlation to improve the overall system performance. 

Recent work in [7] has proposed a characterization of the 
achievable region of orthogonal multiple access schemes with 
correlated sources, based on the computation of joint and 
conditional entropies of the sources, for an arbitrary value 
of N. According to [7], the achievable region, in the space 
of individual channel capacity values is specified by 

the intersection of the following inequalities: 

^A^>ri7(V(5)|X(5e)) (21) 

l^S 

for all S C {1,2, ...,V}, in which V(5) = {Xi : i G 
5} and Sc denotes the complementary set of S. Note that 
H{X{S)\X{Sc)) is the conditional entropy of the sources 
with index in S given the remaining ones. Two characteristic 
operational points, denoted as “balanced” and “unbalanced,” 
are of interest. 

The balanced characteristic point refers to the case where 
all source symbols are transmitted at a rate equal to the same 
single-channel capacity, i.e., Abai = Ai = A 2 = ■ ■ ■ = Xn- 
This common value, is equal to 


Abai = r 


N 


( 22 ) 


The unbalanced case, instead, refers to the portion of the 
achievable region characterized as follows: N—1 sources, e.g.. 
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VI. Conclusions 

In this paper, we have discussed and analyzed a few binary 
source correlation models. In particular, we have statistically 
characterized these models, in terms of joint PMF, covariance, 
and joint entropy of the source correlated sequence. Moreover, 
we have computed the asymptotic entropy rate for large 
number of sources, showing that a common value is obtained 
for all considered models. This result generalizes [7], because 
the asymptotic achievable region of orthogonal multiple access 
schemes for large numbers of sources is shown to be invariant 
to the considered correlation model. Therefore, one can con¬ 
jecture that joint source-channel coded schemes, which exploit 
the correlation at the receiver, may be expected to have similar 
asymptotic behavior regardless of the model. 


Fig. 5. e, as a function of M, for the mixed model, two values of p (namely, 
0.7 and 0.95), and two values of N (namely, 10 and 50). 


sources from 1 to — 1 , are associated with sufficiently large 
values of Ai, i = 1, 2,..., — 1. In this case, Aunb is the 

smallest value of A at such that the operational point lies on 
the border of the achievable region and it is equal to 

Aunb = rH{XN\X^-^). (23) 

In [7], it is shown that for the parallel model in Fig. 0 (a) 
and p£ = p, i = 1^2,... ^ N, the following facts hold 


'^bal ^ -^unb \/N 

lim Aunb = lim Abai = Anm = rHi,{p). (24) 

AT—>-+oo A^—>-+oo 


This characterization of the achievable multiple access region 
is simple but effective. In particular, tells us that, when 
N increases, the achievable region tends to a hyperoctant and 
the system operational points become equal. Therefore, one 
can devise joint source-channel coded schemes for any one 
of these operational points, since they guarantee the same 
achievable rate of other operational points for a sufficiently 
large number of sources. Hence, code design can be based 
on the most convenient operational point, e.g., the one which 
guarantees less complexity. 

A natural question, not discussed in [7], is the following: are 
these results dependent on the considered correlation model? 
In general, the entropy of the binary sources may depend on 
the model considered for correlation. However, we have shown 
in Section IV that the asymptotic entropy rate is the same for 
all the considered models. We can, therefore, conclude that 
the achievable region of orthogonal multiple access schemes 
is asymptotically invariant to the considered correlation model 
as the number of sources increases. Moreover, from Figs. 
and 1 ^ it can be observed that the convergence trend is the 
same for all considered models. Hence, it may be expected 
that joint source-channel coded schemes devised for one of 
these models (see, e.g., the turbo codes considered in [7] for 
the parallel model) have similar asymptotic performance for 
the other models as the number of sources increases. 


Appendix I 

Existence of the Inverse Matrix in the Linear 
Correlation Model @ 

We now present a few special cases of matrix A in (j^ where 
the inverse matrix exists. In particular, we consider two main 
classes: (i) matrices associated with the recursive model in Q 
and (ii) circulant matrices. To show the invertibility of these 
matrices, we can prove that their determinants are non-zero in 
the binary field. 

The coefficients of A for the recursive model in Q are the 
following: 

{ 1 foTk = i 

0 or 1 for k = i — min{iA,— 1},..., f — 1 
0 otherwise 

for i,k = 1,2,..., N. Therefore, the recursive model is 
characterized by a Toeplitz matrix A. It is known that, out 
of all the size-Af Toeplitz matrices over a finite field of q 
elements, a fraction 1 — 1/g (i.e., 1/2 in our binary case) is 
non-singular [12]. The considered matrix associated with the 
recursive model Q is also lower triangular with equal element 
on the main diagonal, since = 0 for i < k. Therefore, the 
determinant can be written as [13]: 

N 

detA = Y\Au = 1. 

r=i 

This proves that the inverse matrix exists for this case. 

Consider now circulant matrices and restrict to those matri¬ 
ces with coefficients, for ^ = 1,2,... ,N, of the form: 

f 1 fov k = ^,^ + 1,... + d — 1 mod W 

Aik = ( 

I 0 otherwise 

where d is such that d consecutive matrix coefficients are 
equal to 1 and the remaining are equal to zero. The mod N 
operation is needed to perform the circular shift of the rows. 
The determinant is known if d is a prime [13]: 

f 0 if d\N 

det A = < 

( d mod 2 otherwise 

where the notation d \ N means that d divides N. Therefore, 
circulant matrices admit an inverse if d is an odd prime (i.e., 
d 7 ^ 2) and does not divide N. 







Appendix II 

Asymptotic Entropy Rate oe the Mixed 
Correlation Model 

Let us consider the joint entropy of X^j and B: 

= H{B) + HiX^i,.. .,X^j^\B). 

(25) 

Given B, the set X(j is independent of X^ for j ^ k and, 
therefore, 

M M N 

\b) = J2h{x^’^\b) = J2Y. Hh{pt])- 

3=1 3=1i=l 

Noting again that B is a uniformly distributed binary random 
variable with H{B) = i7b(0.5) = 1, one obtains; 


M N 


H{B, Xfi,..., X^^) = 1 + 


3 = 1 f-=l 


As in (25 1 , it is also possible to write 


M N 


1+EE HhiPlj) 
3 = 1 t=l 


= H{X 


N 

in • 


H{B,X^,,...,X^m) 

X^^,) + H{B\X^„...,X^m) 


and, therefore, 

M N 

3 = 1 t=l 

(26) 

Since by debnition of entropy the last term is non negative, 
the following UB results 

M N 

H{X^^,X^^) < 1 + E E (27) 

3=1i=l 

Moreover, since conditioning reduces entropy [11], it also 
follows that 

H{B\X^^,.. .,X^j^) < H(BlXn) = ffb(pii). 


Using this in ( |2^ , one obtains the LB; 

M N 

H(X,^„ Xf^) > 1 + 5] L - i7b(pii). (28) 

3 = 1 i=l 

Combining (|27]l and (|28ll, one obtains the limit in (|20ll. 
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